The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
private-mind	Error		Jul 23, 2026 8:46am

RAG improvements — analysis, research & backlog

Analysis of the local-first RAG pipeline on feat/221-rag-sources (attach files → embed on-device → retrieve → cite as sources), a research pass over 2023–2026 literature and the on-device RN/ExecuTorch ecosystem, and a prioritized backlog. The last section tracks what has already been implemented.

Scope of the app this targets: fully on-device (mid-range phones), ExecuTorch .pte models, small LLM (1–4B, ~2048-token context, weak instruction-following), LFM2.5-Embedding-350M (1024-dim), personal-scale corpus (typically 1–20 documents / hundreds–thousands of chunks), Polish + English are first-class.

Evidence caveat: quantitative claims below come from a single research pass (papers were fetched and quoted, but the adversarial cross-check stage did not complete). Treat numbers as directional and validate on a small in-house PL+EN eval set before hard-coding thresholds.

1. What the pipeline does today

Verdict: the retrieval core is strong and in places ahead of common practice (RRF over weighted-sum, MMR, term-coverage boost, Polish diacritics + stem-prefix, neighbor expansion). The weak 10% was the tail: final-set selection, budget packing, and answer-time attribution. Most of that is now addressed (§5) — final-set selection (adaptive-k, per-file cap), budget packing (boundary truncation, leaner instructions), and answer-time attribution (truncation-honest → answer-echo → refusal suppression). Remaining tail work: short-code recall and context de-duplication (§2 A11/A12).

2. Bugs & tensions found in the code (verified with file:line)

These are defects/mismatches, not enhancements — fix first.

• A1 — Prompt budget fights retrieval. Budget = (2048−512)×3 = 4608 chars for the whole prompt (constants/context-window.ts). Retrieval + neighbor expansion + attachment overview + ~1750 chars of appended instructions routinely exceeded it, and utils/promptUtils.ts hard-slice()d the joined context mid-chunk, dropping trailing Source N blocks. ✅ Mitigated (§5): context is ordered most-relevant-first and truncated at a chunk/Source boundary (never mid-passage), and the appended instructions were cut 1750→770 chars — together keeping the answer inside budget. System prompt and question are still never trimmed; the attachment overview is still prepended unconditionally (see §4 [Feature] Implement figma designs #9).
• A2 — Every model treated as 2048 tokens. CONTEXT_WINDOW_TOKENS_BY_FAMILY is {} (constants/context-window.ts:9); the per-family override mechanism exists but is empty. If any .pte export supports 4096+, that budget is left on the table. (Deferred: needs per-model max-seq-len confirmation.)
• A3 — queryText fallback is dead code and swallows keyword search. When the embedder is unavailable, vectorStore.query({queryText}) calls the same embedder and throws; the rejection is inside Promise.all with keywordSearch, so FTS5 results are discarded too and retrieval silently returns []. ✅ Fixed (§5).
• A4 — A fresh attachment fully evicts enabled sources. ATTACHMENT_RELEVANCE_BONUS=+10 guarantees attachment chunks outrank everything; with MAX_RELEVANT_CHUNKS=5 a ≥5-chunk attachment takes all slots, starving other enabled documents ("compare this to last week's contract" can't work). ✅ Mitigated by the per-file cap (§5).
• A5 — Scanned PDF shows "Document appears to be empty." (hooks/useAttachment.ts:203-213) — misleading; the doc isn't empty, it has no text layer. ✅ Fixed (§5).
• A6 — Multi-file attach is structurally impossible though downstream (attachmentSourceIds[], join(', '), prompt) is written for many. Picker lacks multiple:true; setAttachments([...]) replaces the array. (Deferred — see §6.)
• A7 — Source-management UI is dead code. renameSource/deleteSource have no UI caller; a privacy-first app currently can't delete an indexed document. No per-chat sources panel, no disableSource. (Deferred — see §6.)
• A8 — Small-model over-abstention on bare queries (behavioral, from device logs). A 1.7B model under-weights the distant grounding system prompt: a bare question — especially cross-lingual (Polish query over an English doc) — returns "I don't know" even when the answer sits in the retrieved <context>. Typing "według załączonego dokumentu" / "according to the attached document" flips it to a correct answer, confirming grounding, not retrieval, is the blocker — verified with two logs of the same question where the answer chunk led the context in both, yet only the cued one was used. ✅ Mitigated (§5): an always-on grounding cue on the user turn, emitted in the query's language (Polish for a Polish query) and placed after the question, mirroring the proven manual phrase. ⚠️ Confirm on-device that a bare Polish question now answers without the manual "według dokumentu".
• A9 — Citations came from the pre-truncation retrieved set (from device logs). sourceDocuments was built from every retrieved chunk before promptUtils.ts truncated the joined context to budget, so a document dropped to fit the window — or merely co-retrieved from a chat's accumulated enabledSources — was still cited beside the answer ("attach file X, get files Y and Z cited"). Root enabler: every attached file is permanently added to enabledSources (ChatScreen.tsx:223-227) and the chip clears after send (ChatBar.tsx:181), so later turns retrieve across the whole pile. ✅ Fixed (§5): restrictCitationsToContext keeps only sources whose Source N block survived truncation, then pickCitationsByAnswer keeps only those the visible reply echoes.
• A10 — A refusal still cited the sources it described (from device logs). A verbose "no information about L4 in these documents" reply describes both docs to justify the absence, so it lexically overlaps them and slipped past overlap attribution → cited as if grounded. Stripping <think> was not enough (the visible reply itself overlaps). ✅ Fixed (§5): a meta-refusal detector (looksLikeNoAnswer) suppresses all non-attachment citations. Deliberately narrow — a negative-fact answer ("nie ma limitu", "no debt", "does not mention costs") is not a refusal and keeps its citation.
• A11 — Short alphanumeric codes dropped by the tokenizer (from device logs). extractQueryTerms drops tokens < 3 chars (queryTerms.ts:96), so "L4"/"K2"/"E-42" never become query or attribution terms — a document that does contain the code is neither surfaced by keyword search nor attributed to the answer. ⏳ open (§4): pass 2-char letter+digit tokens (codes) while still dropping bare numbers/short words; FTS5 indexing of such codes still to verify.
• A12 — An attachment appears twice in context, confusing the small model (from device logs). buildMessageSources prepends the attachment's overview (Current Attachment Source: … (Overview)) and the same file appears as a retrieved Source N (attachments always clear the gate via the bonus). The 1.7B model then miscounts blocks and reads a --- End of Source N --- closing marker as a phantom source's content. Citations stayed correct, but budget is wasted and the answer's reasoning is muddled. ⏳ open (§4 [Feature] Implement figma designs #9): drop the overview for any attachment that already has a retrieved Source block.

3. Anti-recommendations (deliberately NOT doing — saves weeks)

4. Prioritized backlog

Priority 1 — high-evidence, code-only, no new deps/models/migrations

1. Adaptive-k — stop always sending 5 chunks; cut at the largest gap in raw relevance, 1–5. One related distractor cuts small-model accuracy up to 25% (Power of Noise, SIGIR 2024, incl. Phi-2); quality vs k is an inverted-U (ICLR 2025). ✅ implemented (conservative).
2. Per-file cap + attachment cap — max chunks per document when ≥2 docs are enabled; stops one long doc / one fresh attachment monopolizing the context. ✅ implemented.
3. Prompt reordering — put the best chunk adjacent to the question (worst position today). Lost in the Middle (TACL 2024); replicated at 2.7B. ✅ implemented — expandSelectedWithNeighbors emits relevance-ranked windows (the best seed's [prev, seed, next] first), so a budget truncation drops the least-relevant tail rather than the answer. Source↔citation alignment held: getSourceDocumentsFromChunks groups the same chunks.
4. Follow-up query contextualization (tier-1, no LLM) — under-specified queries ("a ile to kosztowało?") collapse below the gate; concatenate prior turn + last-answer terms into the retrieval query only. Biggest real multi-turn failure mode. ⏳ deferred (needs call-site history threading).
5. Overlap-dedup at stitching — neighbor expansion re-duplicates the 200-char overlap in the prompt; strip it. ✅ implemented.
6. Contextual chunk headers from metadata (no LLM) — prepend Dokument: {name} | Strona {n} | Sekcja: {…} before embedding + FTS; the file name is invisible to both retrievers today. Anthropic Contextual Retrieval −49% failure rate; heading-breadcrumb variant gets most of it for free. ⏳ deferred (re-index migration).
7. FTS5 for Polish — emit ("kosztowało" OR "kosztowa"*) instead of replacing exact form with prefix; add a phrase-match retriever; declare prefix='4 5 6'. BEIR-PL: BM25 degrades on Polish inflection. ⏳ deferred (re-index).
8. Retrieval status UX — after send, the typed text vanishes and nothing shows during retrieval; render "Przeszukuję dokumenty…". ⏳ deferred (UI/store change).
9. Conditional attachment overview (fixes A12) — the cover-page first chunk is prepended for every attachment turn (messageSources.ts), spending ~1000 chars of budget on boilerplate even for pointed factual questions, and duplicating a file that also appears as a retrieved Source N — which confuses the small model into a phantom source (A12). Inject the overview only when retrieval surfaced none of the attachment's own chunks (broad / "summarize this" fallback), and trail the retrieved chunks so truncation drops it first. ⏳ open (needs care: the overview is the only signal when a broad query retrieves nothing).
10. Short-code tokenization (fixes A11) — let extractQueryTerms keep 2-char letter+digit tokens ("L4", "K2", "3D") while still dropping bare numbers and short words, so a code the user asks about drives keyword search + attribution. Shared tokenizer → also improves the citation highlight. ⏳ open (verify FTS5 indexes such codes; keep single-file precision).

Priority 2 — medium cost, high value

• Post-hoc citation attribution (CiteFix-style) — ✅ implemented (§5), keyword/term-overlap variant: cite a source only if the visible reply echoes its passage terms (≥ ratio of the strongest), with <think> stripped and a meta-refusal citing nothing. CiteFix (ACL 2025 Industry): +15.5% citation accuracy, 15 ms. Still open: the cosine blend (~0.8·keyword + 0.2·cosine, chunk embeddings already in memory) and the answer-derived highlight + tappable chips → SourcesSheet.present(index) (the index param already exists and is unused).
• Zero-model groundedness badge — stopword-filtered answer↔context token overlap (K-Precision). TACL 2024: Spearman ~0.50, beats NLI (~0.29) and BERTScore (~0.23); do NOT build it on embedding cosine. Soft badge only.
• Pre-generation abstention — when the gate passes nothing, skip the LLM, show "Nie znalazłem tego w dokumentach" + "Answer without documents". Small models can't follow "say I don't know" (Sufficient Context, ICLR 2025).
• Per-query gate normalization — replace the fixed cosine ≥ 0.55 with a percentile/z-score cut. Cosine scores aren't comparable across queries (Cosine Adapter, CIKM 2024); 0.55 likely over-filters Polish (not among LFM2.5's 11 languages). Needs a small eval set.
• Page-aware PDF — the vendored pdfium extractText() already loops per page; emit a page delimiter → pageStart in metadata → "p. 12" citations + section headers from bookmarks.
• Multi-file attach — multiple:true, append not replace, sequential ingest queue, cap ~5.
• DOCX via mammoth.js browser build ({arrayBuffer} from Expo File API, convertToHtml→markdown).
• Resumable ingestion — insert all chunks (embedding=NULL) first, embed via an idempotent WHERE embedding IS NULL cursor; survives app kill.
• int8 embedding quantization (+ optional MRL→512) — ~99% nDCG retained at 4× storage; LFM2.5 is Matryoshka-trained. Format migration, no re-embed.
• Per-chat sources panel + library reuse — wire the existing enabledSources backend to toggles + "add from library" (vectors already exist → zero re-embed) + a global rename/delete screen.

Priority 3 — roadmap

• ONNX cross-encoder reranker routed by language (PL: sdadas/polish-reranker-base-ranknet ~124M; EN: ms-marco-MiniLM 22M) via onnxruntime-react-native, behind a setting.
• On-device OCR (ML Kit v2 Latin covers Polish; Apple Vision does not support Polish) for scanned PDFs/images.
• On-device grounding verifier (LettuceDetect EuroBERT-210M or CiteVerifier 355M — both cover Polish; skip generic mDeBERTa-NLI).
• KV prefix caching of the static instruction prefix once the runtime exposes it (Prompt Cache, MLSys 2024: up to 60× TTFT on CPU).

5. Implemented in this pass

Code-only, no new dependencies, no native changes, no re-index/data migration. All covered by unit tests.

Compact hot-path logs (RAG retrieved / RAG prepared / LLM done with per-doc attribution, noAnswer, and a short visible-reply preview) make citation decisions diagnosable from the terminal without dumping document text.

Tuning constants live in constants/retrieval.ts (MAX_CHUNKS_PER_FILE, ADAPTIVE_K_DROP_RATIO, ADAPTIVE_K_MIN_KEEP, ANSWER_CITATION_OVERLAP_RATIO) so the above can be tuned once an eval set exists.